Summary and discussion of: “Dropout Training as Adaptive Regularization”

Multi-layered (i.e. deep) artificial neural networks have recently undergone a resurgence in popularity due to improved processing capabilities and the increasing availability of large datasets. Popular in the 1980’s and before, they were largely abandoned in favor of convex methods (such as support vector machines) that came with optimality guarantees and often gave better results in far less time. However, with the modern ubiquity of GPUs and distributed computing, the same methods that were once spurned for their computational intractability have become the de-facto standard for large-scale commercial applications in companies such as Google, Facebook, Baidu, etc. This almost universal adoption of deep learning (as it’s now called) is not without reason; variants of these methods have achieved state-of-the-art performance (often by a significant margin over other competing algorithms) on numerous tasks such as image classification, speech recognition, etc. While improved computational power has allowed these models to be trained in a moderate amount of time, their performance is tied to the quantity and quality of the available training data. Deep neural networks are often very large, often consisting upwards of millions (or even billions) of parameters whose values must all be learnt from data. Thus, like any statistical model, they are susceptible to overfitting and typically require huge quantities of labeled training examples. Furthermore, the optimization landscape for the network parameters is highly non-convex with many local optima. So even with sufficient training data, the performance of a trained model could still be poor. These problems have been addressed empirically by a surprisingly simple method referred to as dropout. First introduced by Geoffrey Hinton, dropout has become an important component in most modern deep learning implementations because it results in reduced overfitting and often reaches better local minima. Even so, very little is understood theoretically about why this should be the case. The paper Dropout Training as Adaptive Regularization is one of several recent papers that attempts to understand the role of dropout in training deep neural networks.